Problem Statement

Wind energy has emerged as one of the most promising renewable energy sources worldwide. However, the operation and maintenance of wind turbines are costly, especially when failures occur unexpectedly. Traditional maintenance practices either schedule repairs too frequently (leading to unnecessary costs) or too late (resulting in catastrophic breakdowns and replacements).

ReneWind aims to leverage machine learning and predictive maintenance to accurately identify potential generator failures in wind turbines. By analyzing sensor data, the company seeks to predict failures before they happen, thereby reducing downtime, minimizing costs, and improving operational efficiency.

Objective

  • The primary objective is to develop, tune and evaluate models that can:
    • Accurately predict wind turbine generator failures (failures = 1, no failures = 0).
    • Maximize detection of true failures (high recall) while minimizing false alarms (high precision).
    • Support predictive maintenance by ensuring components are repaired before breakdowns.
    • Reduce overall operation and maintenance costs by minimizing:
      • Replacement costs (from missed failures - false negatives).
      • Repair costs (from correctly predicted failures - true positives). Inspection costs (from false alarms - false positives).
    • Provide a scalable and reliable framework that can be integrated into real-time turbine monitoring systems.

Data Dictionary

  • The data provided is a transformed version of the original data which was collected using sensors.

    • Train.csv - To be used for training and tuning of models (20,000 observations).
    • Test.csv - To be used only for testing the performance of the final best model (5,000 observations).
  • Both datasets consist of 40 predictor variables and 1 target variable.

  • Target Variable:
    • 0 - No Failure (turbine operates normally).
    • 1 - Failure (turbine experiences a generator failure).

Importing necessary libraries

In [69]:
# importing core libraries for data manipulation and visualisation
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# importing scikit-learn utilities for preprocessing, model building and evaluation
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, f1_score, roc_auc_score, precision_score, recall_score
from sklearn.utils.class_weight import compute_class_weight

# to oversample and undersample data
from imblearn.over_sampling import SMOTE
from imblearn.over_sampling import RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler

# importing tensorflow utilities
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import SGD, Adam
from tensorflow.keras.callbacks import EarlyStopping

import warnings
warnings.filterwarnings("ignore")

Loading the data

In [70]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [71]:
# loading data into a pandas dataframe
df_train = pd.read_csv("/content/drive/MyDrive/GL_Project/ReneWind/Train.csv")
df_test = pd.read_csv("/content/drive/MyDrive/GL_Project/ReneWind/Test.csv")
In [72]:
# creating copy of the data
df_train_copy = df_train.copy()
df_test_copy = df_test.copy()

Data overview

In [73]:
# displaying first 5 rows of the train data
df_train.head(5)
Out[73]:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ... V32 V33 V34 V35 V36 V37 V38 V39 V40 Target
0 -4.464606 -4.679129 3.101546 0.506130 -0.221083 -2.032511 -2.910870 0.050714 -1.522351 3.761892 ... 3.059700 -1.690440 2.846296 2.235198 6.667486 0.443809 -2.369169 2.950578 -3.480324 0
1 3.365912 3.653381 0.909671 -1.367528 0.332016 2.358938 0.732600 -4.332135 0.565695 -0.101080 ... -1.795474 3.032780 -2.467514 1.894599 -2.297780 -1.731048 5.908837 -0.386345 0.616242 0
2 -3.831843 -5.824444 0.634031 -2.418815 -1.773827 1.016824 -2.098941 -3.173204 -2.081860 5.392621 ... -0.257101 0.803550 4.086219 2.292138 5.360850 0.351993 2.940021 3.839160 -4.309402 0
3 1.618098 1.888342 7.046143 -1.147285 0.083080 -1.529780 0.207309 -2.493629 0.344926 2.118578 ... -3.584425 -2.577474 1.363769 0.622714 5.550100 -1.526796 0.138853 3.101430 -1.277378 0
4 -0.111440 3.872488 -3.758361 -2.982897 3.792714 0.544960 0.205433 4.848994 -1.854920 -6.220023 ... 8.265896 6.629213 -10.068689 1.222987 -3.229763 1.686909 -2.163896 -3.644622 6.510338 0

5 rows × 41 columns

Observation:

The first 5 rows of the train data along with the title of the column has been displayed here.

In [74]:
# displaying first 5 rows of the test data
df_test.head(5)
Out[74]:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ... V32 V33 V34 V35 V36 V37 V38 V39 V40 Target
0 -0.613489 -3.819640 2.202302 1.300420 -1.184929 -4.495964 -1.835817 4.722989 1.206140 -0.341909 ... 2.291204 -5.411388 0.870073 0.574479 4.157191 1.428093 -10.511342 0.454664 -1.448363 0
1 0.389608 -0.512341 0.527053 -2.576776 -1.016766 2.235112 -0.441301 -4.405744 -0.332869 1.966794 ... -2.474936 2.493582 0.315165 2.059288 0.683859 -0.485452 5.128350 1.720744 -1.488235 0
2 -0.874861 -0.640632 4.084202 -1.590454 0.525855 -1.957592 -0.695367 1.347309 -1.732348 0.466500 ... -1.318888 -2.997464 0.459664 0.619774 5.631504 1.323512 -1.752154 1.808302 1.675748 0
3 0.238384 1.458607 4.014528 2.534478 1.196987 -3.117330 -0.924035 0.269493 1.322436 0.702345 ... 3.517918 -3.074085 -0.284220 0.954576 3.029331 -1.367198 -3.412140 0.906000 -2.450889 0
4 5.828225 2.768260 -1.234530 2.809264 -1.641648 -1.406698 0.568643 0.965043 1.918379 -2.774855 ... 1.773841 -1.501573 -2.226702 4.776830 -6.559698 -0.805551 -0.276007 -3.858207 -0.537694 0

5 rows × 41 columns

Observation:

The first 5 rows of the test data along with the title of the column has been displayed here.

In [75]:
# checking the shape of the train data
df_train.shape
Out[75]:
(20000, 41)

Observation:

The data.shape is used to generate the number of rows and columns. Here, the train dataset has 20000 rows and 41 columns.

In [76]:
# checking the shape of the test data
df_test.shape
Out[76]:
(5000, 41)

Observation:

The data.shape is used to generate the number of rows and columns. Here, the train dataset has 50000 rows and 41 columns.

In [77]:
# checking the attribute types of the train data
df_train.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20000 entries, 0 to 19999
Data columns (total 41 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   V1      19982 non-null  float64
 1   V2      19982 non-null  float64
 2   V3      20000 non-null  float64
 3   V4      20000 non-null  float64
 4   V5      20000 non-null  float64
 5   V6      20000 non-null  float64
 6   V7      20000 non-null  float64
 7   V8      20000 non-null  float64
 8   V9      20000 non-null  float64
 9   V10     20000 non-null  float64
 10  V11     20000 non-null  float64
 11  V12     20000 non-null  float64
 12  V13     20000 non-null  float64
 13  V14     20000 non-null  float64
 14  V15     20000 non-null  float64
 15  V16     20000 non-null  float64
 16  V17     20000 non-null  float64
 17  V18     20000 non-null  float64
 18  V19     20000 non-null  float64
 19  V20     20000 non-null  float64
 20  V21     20000 non-null  float64
 21  V22     20000 non-null  float64
 22  V23     20000 non-null  float64
 23  V24     20000 non-null  float64
 24  V25     20000 non-null  float64
 25  V26     20000 non-null  float64
 26  V27     20000 non-null  float64
 27  V28     20000 non-null  float64
 28  V29     20000 non-null  float64
 29  V30     20000 non-null  float64
 30  V31     20000 non-null  float64
 31  V32     20000 non-null  float64
 32  V33     20000 non-null  float64
 33  V34     20000 non-null  float64
 34  V35     20000 non-null  float64
 35  V36     20000 non-null  float64
 36  V37     20000 non-null  float64
 37  V38     20000 non-null  float64
 38  V39     20000 non-null  float64
 39  V40     20000 non-null  float64
 40  Target  20000 non-null  int64  
dtypes: float64(40), int64(1)
memory usage: 6.3 MB

Observation:

  • There is 1 column with integer (int64) values (Target).
  • There are 40 columns with float (float64) value.
In [78]:
# checking the attribute types of the test data
df_test.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 41 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   V1      4995 non-null   float64
 1   V2      4994 non-null   float64
 2   V3      5000 non-null   float64
 3   V4      5000 non-null   float64
 4   V5      5000 non-null   float64
 5   V6      5000 non-null   float64
 6   V7      5000 non-null   float64
 7   V8      5000 non-null   float64
 8   V9      5000 non-null   float64
 9   V10     5000 non-null   float64
 10  V11     5000 non-null   float64
 11  V12     5000 non-null   float64
 12  V13     5000 non-null   float64
 13  V14     5000 non-null   float64
 14  V15     5000 non-null   float64
 15  V16     5000 non-null   float64
 16  V17     5000 non-null   float64
 17  V18     5000 non-null   float64
 18  V19     5000 non-null   float64
 19  V20     5000 non-null   float64
 20  V21     5000 non-null   float64
 21  V22     5000 non-null   float64
 22  V23     5000 non-null   float64
 23  V24     5000 non-null   float64
 24  V25     5000 non-null   float64
 25  V26     5000 non-null   float64
 26  V27     5000 non-null   float64
 27  V28     5000 non-null   float64
 28  V29     5000 non-null   float64
 29  V30     5000 non-null   float64
 30  V31     5000 non-null   float64
 31  V32     5000 non-null   float64
 32  V33     5000 non-null   float64
 33  V34     5000 non-null   float64
 34  V35     5000 non-null   float64
 35  V36     5000 non-null   float64
 36  V37     5000 non-null   float64
 37  V38     5000 non-null   float64
 38  V39     5000 non-null   float64
 39  V40     5000 non-null   float64
 40  Target  5000 non-null   int64  
dtypes: float64(40), int64(1)
memory usage: 1.6 MB

Observation:

  • There is 1 column with integer (int64) values (Target).
  • There are 40 columns with float (float64) value.

Univariate Analysis

In [79]:
# countplot for target variable distribution

sns.countplot(x = 'Target', data = df_train, palette = 'Set2')
plt.title('Distribution of Target Variable')
plt.show()

Observation:

  • The plot shows high imbalance in the data:

    * Class 0 (No failures) has around 19000 samples.
    * Class 1 (Failures) has around 2000 samples.
  • The ratio seems to be 90:10 with the Class 1 as minority class.

In [80]:
# univariate analysis
target_col = 'Target'
features = [col for col in df_train.columns if col != target_col]   # to seprate features
for col in features:
  plt.figure(figsize=(14,4))

  # histogram
  plt.subplot(1,3,1)
  sns.distplot(df_train[col], kde=True, bins=30, color='red')
  plt.title(f'Histogram distribution of {col}')

  # kde
  plt.subplot(1,3,2)
  sns.kdeplot(df_train[col], shade=True, color='skyblue')
  plt.title(f'KDE distribution of {col}')

  # boxplot
  plt.subplot(1,3,3)
  sns.boxplot(y=df_train[col], color='lightgreen')
  plt.title(f'Boxplot distribution of {col}')

  plt.show()

Observation:

  • The histogram seems to have a normal distribution with very few values beyond +10/-10.
  • The KDE distribution is symmetric and has its distribution close to Gaussian.
  • The strong density of the KDE distribution shows that the graph is normally distributed without strong skewness.
  • The box plot shows that the meadian is 0 and the interquartile range (IQR) is between -2 to 2.
  • Also, the box plot shows that the outliers exists but are sparse.
In [81]:
# bivariate analysis
for col in features:
  plt.figure(figsize=(12,4))

# boxplot
plt.subplot(1,2,1)
sns.boxplot(x='Target', y=col, data=df_train, palette='Set2')
plt.title(f'Boxplot distribution of {col} vs Target')

# violin plot
plt.subplot(1,2,2)
sns.violinplot(x='Target', y=col, data=df_train, palette='muted')
plt.title(f'Violin plot distribution of {col} vs Target')

# heatmap
plt.figure(figsize=(15,10))
sns.heatmap(df_train.corr(), annot=True, cmap='coolwarm')
plt.title('Correlation Heatmap')

# pairplot
ss = features[:5] + [target_col]
sns.pairplot(df_train[ss], hue='Target')
plt.title('Pairplot of Features')

plt.show()
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>
<Figure size 1200x400 with 0 Axes>

Observation:

  • The box plot and violin plot for the V40 vs Target distribution shows that there is no strong sepration between the two Target classes.
  • The density in violin plot is high around values close to zero for both the classes.
  • In heatmap, the strong correlations (~ +1) appears in red while the strong negative correlations (~ -1) appears in blue.
  • Also, heatmap shows that the features are not strongly correlated with the target; some are highly correlated with each other.
  • The pairplot shows that the diagonal plots are roughly symmetric and has Gaussian distribution.
  • In pairplot, the orange points (Target = 1) are less frequent and also the overall classes are overlapping a lot.

Data Preprocessing

In [82]:
# checking statistcal summary of train data

df_train.describe()
Out[82]:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ... V32 V33 V34 V35 V36 V37 V38 V39 V40 Target
count 19982.000000 19982.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 ... 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000 20000.000000
mean -0.271996 0.440430 2.484699 -0.083152 -0.053752 -0.995443 -0.879325 -0.548195 -0.016808 -0.012998 ... 0.303799 0.049825 -0.462702 2.229620 1.514809 0.011316 -0.344025 0.890653 -0.875630 0.055500
std 3.441625 3.150784 3.388963 3.431595 2.104801 2.040970 1.761626 3.295756 2.160568 2.193201 ... 5.500400 3.575285 3.183841 2.937102 3.800860 1.788165 3.948147 1.753054 3.012155 0.228959
min -11.876451 -12.319951 -10.708139 -15.082052 -8.603361 -10.227147 -7.949681 -15.657561 -8.596313 -9.853957 ... -19.876502 -16.898353 -17.985094 -15.349803 -14.833178 -5.478350 -17.375002 -6.438880 -11.023935 0.000000
25% -2.737146 -1.640674 0.206860 -2.347660 -1.535607 -2.347238 -2.030926 -2.642665 -1.494973 -1.411212 ... -3.420469 -2.242857 -2.136984 0.336191 -0.943809 -1.255819 -2.987638 -0.272250 -2.940193 0.000000
50% -0.747917 0.471536 2.255786 -0.135241 -0.101952 -1.000515 -0.917179 -0.389085 -0.067597 0.100973 ... 0.052073 -0.066249 -0.255008 2.098633 1.566526 -0.128435 -0.316849 0.919261 -0.920806 0.000000
75% 1.840112 2.543967 4.566165 2.130615 1.340480 0.380330 0.223695 1.722965 1.409203 1.477045 ... 3.761722 2.255134 1.436935 4.064358 3.983939 1.175533 2.279399 2.057540 1.119897 0.000000
max 15.493002 13.089269 17.090919 13.236381 8.133797 6.975847 8.006091 11.679495 8.137580 8.108472 ... 23.633187 16.692486 14.358213 15.291065 19.329576 7.467006 15.289923 7.759877 10.654265 1.000000

8 rows × 41 columns

Observation:

The train data has minimum feature value is around -3.42 while the maximum feature value is around 23.63.

In [83]:
# checking statistcal summary of test data

df_test.describe()
Out[83]:
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 ... V32 V33 V34 V35 V36 V37 V38 V39 V40 Target
count 4995.000000 4994.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 ... 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000
mean -0.277622 0.397928 2.551787 -0.048943 -0.080120 -1.042138 -0.907922 -0.574592 0.030121 0.018524 ... 0.232567 -0.080115 -0.392663 2.211205 1.594845 0.022931 -0.405659 0.938800 -0.932406 0.056400
std 3.466280 3.139562 3.326607 3.413937 2.110870 2.005444 1.769017 3.331911 2.174139 2.145437 ... 5.585628 3.538624 3.166101 2.948426 3.774970 1.785320 3.968936 1.716502 2.978193 0.230716
min -12.381696 -10.716179 -9.237940 -14.682446 -7.711569 -8.924196 -8.124230 -12.252731 -6.785495 -8.170956 ... -17.244168 -14.903781 -14.699725 -12.260591 -12.735567 -5.079070 -15.334533 -5.451050 -10.076234 0.000000
25% -2.743691 -1.649211 0.314931 -2.292694 -1.615238 -2.368853 -2.054259 -2.642088 -1.455712 -1.353320 ... -3.556267 -2.348121 -2.009604 0.321818 -0.866066 -1.240526 -2.984480 -0.208024 -2.986587 0.000000
50% -0.764767 0.427369 2.260428 -0.145753 -0.131890 -1.048571 -0.939695 -0.357943 -0.079891 0.166292 ... -0.076694 -0.159713 -0.171745 2.111750 1.702964 -0.110415 -0.381162 0.959152 -1.002764 0.000000
75% 1.831313 2.444486 4.587000 2.166468 1.341197 0.307555 0.212228 1.712896 1.449548 1.511248 ... 3.751857 2.099160 1.465402 4.031639 4.104409 1.237522 2.287998 2.130769 1.079738 0.000000
max 13.504352 14.079073 15.314503 12.140157 7.672835 5.067685 7.616182 10.414722 8.850720 6.598728 ... 26.539391 13.323517 12.146302 13.489237 17.116122 6.809938 13.064950 7.182237 8.698460 1.000000

8 rows × 41 columns

Observation:

The test data has minimum feature value is around -17.24 while the maximum feature value is around 26.53.

In [84]:
# identifying target and features
X = df_train.drop('Target', axis=1)
y = df_train['Target'].copy()

X_test = df_test.drop(['Target'], axis=1)
y_test = df_test['Target'].copy()
In [85]:
# splitting data into train and validation to preprocess it
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

print(f"X_train shape: {X_train.shape}")
print(f"y_train shape: {y_train.shape}")
print(f"X_val shape: {X_val.shape}")
print(f"y_val shape: {y_val.shape}")
print(f"X_test shape: {X_test.shape}")
print(f"y_test shape: {y_test.shape}")
X_train shape: (16000, 40)
y_train shape: (16000,)
X_val shape: (4000, 40)
y_val shape: (4000,)
X_test shape: (5000, 40)
y_test shape: (5000,)

Observation:

  • The train data has 16000 samples and 40 features.
  • The validation data has 4000 samples and 40 features.
  • The test data has 5000 samples and 40 features.
In [86]:
# creating an instace of the imputer to be used
imputer = SimpleImputer(strategy="median")
In [87]:
# fit and transform the train data
X_train = pd.DataFrame(imputer.fit_transform(X_train), columns=X_train.columns)

# Transform the validation data
X_val =  pd.DataFrame(imputer.transform(X_val), columns=X_train.columns)

# Transform the test data
X_test = pd.DataFrame(imputer.transform(X_test), columns=X_train.columns)
In [88]:
# checking missing values

print(f"Missing values in X_train:\n{X_train.isnull().sum()}")
print(f"\nMissing values in X_val:\n{X_val.isnull().sum()}")
print(f"\nMissing values in X_test:\n{X_test.isnull().sum()}")
Missing values in X_train:
V1     0
V2     0
V3     0
V4     0
V5     0
V6     0
V7     0
V8     0
V9     0
V10    0
V11    0
V12    0
V13    0
V14    0
V15    0
V16    0
V17    0
V18    0
V19    0
V20    0
V21    0
V22    0
V23    0
V24    0
V25    0
V26    0
V27    0
V28    0
V29    0
V30    0
V31    0
V32    0
V33    0
V34    0
V35    0
V36    0
V37    0
V38    0
V39    0
V40    0
dtype: int64

Missing values in X_val:
V1     0
V2     0
V3     0
V4     0
V5     0
V6     0
V7     0
V8     0
V9     0
V10    0
V11    0
V12    0
V13    0
V14    0
V15    0
V16    0
V17    0
V18    0
V19    0
V20    0
V21    0
V22    0
V23    0
V24    0
V25    0
V26    0
V27    0
V28    0
V29    0
V30    0
V31    0
V32    0
V33    0
V34    0
V35    0
V36    0
V37    0
V38    0
V39    0
V40    0
dtype: int64

Missing values in X_test:
V1     0
V2     0
V3     0
V4     0
V5     0
V6     0
V7     0
V8     0
V9     0
V10    0
V11    0
V12    0
V13    0
V14    0
V15    0
V16    0
V17    0
V18    0
V19    0
V20    0
V21    0
V22    0
V23    0
V24    0
V25    0
V26    0
V27    0
V28    0
V29    0
V30    0
V31    0
V32    0
V33    0
V34    0
V35    0
V36    0
V37    0
V38    0
V39    0
V40    0
dtype: int64

Observation:

There are no missing values in the given dataset of ReneWind.

In [89]:
# checking for duplicate values

df_train.duplicated().sum()
Out[89]:
np.int64(0)

Observation:

There are no duplicate values in the given dataset of ReneWind.

In [90]:
# checking for outliers
for col in features[:5]:
  plt.figure(figsize=(6,4))
  sns.boxplot(x=df_train[col], color='lightgreen')
  plt.title(f'Boxplot distribution of {col}')
  plt.show()

out_sum = {}
for col in features:
  if pd.api.types.is_numeric_dtype(df_train[col]):  # only numeric features
        Q1 = df_train[col].quantile(0.25)
        Q3 = df_train[col].quantile(0.75)
        IQR = Q3 - Q1
        lower_bound = Q1 - 1.5 * IQR
        upper_bound = Q3 + 1.5 * IQR
        outliers = ((df_train[col] < lower_bound) | (df_train[col] > upper_bound)).sum()
        out_sum[col] = outliers

Observation:

  • The distribution of features shows that the given features has outliers.
  • V1 and V3 seems to be right-skewed as more extreme values are on the positive side.
  • V2 and V5 distribution seems to be symmetric with balanced outliers.
  • V4 is left-skewed has it has more negative outliers.

Model Building

In [91]:
# adding scale features

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)
X_test_scaled = scaler.transform(X_test)

Model evaluation criterion

The nature of predictions made by the classification model will translate as follows:

  • True positives (TP) are failures correctly predicted by the model. These will result in repair costs.
  • False negatives (FN) are real failures where there is no detection by the model. These will result in replacement costs.
  • False positives (FP) are detections where there is no failure. These will result in inspection costs.
In [92]:
# model evaluation

def evaluate_model(model, X, y, set_name="Set"):

  X = np.array(X)
  y = np.array(y)

  prob = model.predict(X).ravel()
  preds = (prob > 0.5).astype(int)

  acc = accuracy_score(y, preds)
  f1 = f1_score(y, preds)
  roc = roc_auc_score(y, preds)
  prec = precision_score(y, preds)
  rec = recall_score(y, preds)

  # try/except for ROC curve
  try:
    roc = roc_auc_score(y, prob)
  except Exception:
    roc = roc_auc_score(y, preds)

  print(f"\tEvaluation on {set_name}\n")
  print(f"Accuracy       : {acc:.4f}")
  print(f"F1 Score       : {f1:.4f}")
  print(f"ROC AUC Score  : {roc:.4f}")
  print(f"Precision      : {prec:.4f}")
  print(f"Recall         : {rec:.4f}")
  print("\nConfusion Matrix:\n", confusion_matrix(y, preds))
  print("\nClassification Report:\n", classification_report(y, preds))
  return {"accuracy": acc, "f1": f1, "roc_auc": roc, "precision": prec, "recall": rec}

Model Performance Improvement and Final Model Selection

In [93]:
# computing class weights for imbalance

class_weights_arr = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
class_weights = {i: w for i, w in enumerate(class_weights_arr)}
print("Class weights:", class_weights)
Class weights: {0: np.float64(0.5293806246691372), 1: np.float64(9.00900900900901)}
In [94]:
# building model

def build_model(layers, optimizer, dropout_rate=0.0, X_tr=None, y_tr=None, X_val=None, y_val=None, use_class_weights=False, epochs=50, batch_size=64, verbose=1):
  modell = Sequential()

  # input layer
  modell.add(Dense(layers[0], activation='relu', input_dim = X_tr.shape[1]))

  # hidden layer
  for size in layers[1:]:
   modell.add(Dense(size, activation='relu'))
   if dropout_rate > 0:
      modell.add(Dropout(dropout_rate))

  modell.add(Dense(1, activation='sigmoid'))     # output layer
  modell.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])    # compiling model

  # callbacks
  es = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
  print(f"\nTraining model: layers={layers}, optimizer={type(optimizer.name).__name__}, dropout={dropout_rate}, class_weights={use_class_weights}")

  hist = modell.fit(X_tr, y_tr,
                   epochs=epochs, batch_size=64,
                   validation_data=(X_val, y_val),
                   verbose=verbose,
                   class_weight = class_weights if use_class_weights else None)

  # evaluating model
  ev_tr = evaluate_model(modell, X_tr, y_tr, set_name="Training")
  ev_va = evaluate_model(modell, X_val, y_val, set_name="Validation")
  return modell, hist
In [95]:
# model 1: baseline - shallow NN + SGD
m1, h1 = build_model([32, 16], SGD(learning_rate=0.01),
                     dropout_rate=0.0, X_tr=X_train_scaled, y_tr = y_train.values,
                     X_val = X_val_scaled, y_val = y_val.values,
                     use_class_weights=False, epochs=50, batch_size=64, verbose=1)
Training model: layers=[32, 16], optimizer=str, dropout=0.0, class_weights=False
Epoch 1/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.8870 - loss: 0.3738 - val_accuracy: 0.9445 - val_loss: 0.2118
Epoch 2/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - accuracy: 0.9475 - loss: 0.1905 - val_accuracy: 0.9517 - val_loss: 0.1739
Epoch 3/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9531 - loss: 0.1619 - val_accuracy: 0.9592 - val_loss: 0.1525
Epoch 4/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9607 - loss: 0.1433 - val_accuracy: 0.9630 - val_loss: 0.1361
Epoch 5/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9653 - loss: 0.1300 - val_accuracy: 0.9660 - val_loss: 0.1231
Epoch 6/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9696 - loss: 0.1193 - val_accuracy: 0.9697 - val_loss: 0.1129
Epoch 7/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9729 - loss: 0.1077 - val_accuracy: 0.9730 - val_loss: 0.1047
Epoch 8/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9747 - loss: 0.0964 - val_accuracy: 0.9743 - val_loss: 0.0981
Epoch 9/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9730 - loss: 0.0966 - val_accuracy: 0.9772 - val_loss: 0.0928
Epoch 10/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9757 - loss: 0.0918 - val_accuracy: 0.9803 - val_loss: 0.0881
Epoch 11/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9806 - loss: 0.0819 - val_accuracy: 0.9818 - val_loss: 0.0842
Epoch 12/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9784 - loss: 0.0853 - val_accuracy: 0.9822 - val_loss: 0.0810
Epoch 13/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9807 - loss: 0.0817 - val_accuracy: 0.9835 - val_loss: 0.0783
Epoch 14/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9820 - loss: 0.0743 - val_accuracy: 0.9837 - val_loss: 0.0762
Epoch 15/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9839 - loss: 0.0725 - val_accuracy: 0.9843 - val_loss: 0.0740
Epoch 16/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9830 - loss: 0.0738 - val_accuracy: 0.9845 - val_loss: 0.0722
Epoch 17/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9832 - loss: 0.0753 - val_accuracy: 0.9847 - val_loss: 0.0705
Epoch 18/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9857 - loss: 0.0682 - val_accuracy: 0.9850 - val_loss: 0.0690
Epoch 19/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9859 - loss: 0.0697 - val_accuracy: 0.9858 - val_loss: 0.0676
Epoch 20/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9864 - loss: 0.0639 - val_accuracy: 0.9862 - val_loss: 0.0664
Epoch 21/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9862 - loss: 0.0673 - val_accuracy: 0.9870 - val_loss: 0.0653
Epoch 22/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9850 - loss: 0.0719 - val_accuracy: 0.9870 - val_loss: 0.0642
Epoch 23/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9858 - loss: 0.0643 - val_accuracy: 0.9865 - val_loss: 0.0632
Epoch 24/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9859 - loss: 0.0632 - val_accuracy: 0.9877 - val_loss: 0.0622
Epoch 25/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9878 - loss: 0.0571 - val_accuracy: 0.9877 - val_loss: 0.0613
Epoch 26/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9860 - loss: 0.0652 - val_accuracy: 0.9870 - val_loss: 0.0604
Epoch 27/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9889 - loss: 0.0521 - val_accuracy: 0.9885 - val_loss: 0.0598
Epoch 28/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9875 - loss: 0.0605 - val_accuracy: 0.9875 - val_loss: 0.0589
Epoch 29/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9884 - loss: 0.0548 - val_accuracy: 0.9885 - val_loss: 0.0582
Epoch 30/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9872 - loss: 0.0600 - val_accuracy: 0.9883 - val_loss: 0.0576
Epoch 31/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9887 - loss: 0.0580 - val_accuracy: 0.9885 - val_loss: 0.0570
Epoch 32/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9898 - loss: 0.0492 - val_accuracy: 0.9885 - val_loss: 0.0565
Epoch 33/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9887 - loss: 0.0500 - val_accuracy: 0.9890 - val_loss: 0.0560
Epoch 34/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9882 - loss: 0.0524 - val_accuracy: 0.9893 - val_loss: 0.0556
Epoch 35/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9876 - loss: 0.0576 - val_accuracy: 0.9893 - val_loss: 0.0548
Epoch 36/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9899 - loss: 0.0535 - val_accuracy: 0.9900 - val_loss: 0.0548
Epoch 37/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9884 - loss: 0.0574 - val_accuracy: 0.9898 - val_loss: 0.0540
Epoch 38/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9888 - loss: 0.0542 - val_accuracy: 0.9900 - val_loss: 0.0539
Epoch 39/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9893 - loss: 0.0517 - val_accuracy: 0.9900 - val_loss: 0.0535
Epoch 40/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9890 - loss: 0.0508 - val_accuracy: 0.9900 - val_loss: 0.0531
Epoch 41/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9900 - loss: 0.0470 - val_accuracy: 0.9900 - val_loss: 0.0526
Epoch 42/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9901 - loss: 0.0479 - val_accuracy: 0.9902 - val_loss: 0.0523
Epoch 43/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9873 - loss: 0.0588 - val_accuracy: 0.9898 - val_loss: 0.0519
Epoch 44/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9886 - loss: 0.0560 - val_accuracy: 0.9910 - val_loss: 0.0518
Epoch 45/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9893 - loss: 0.0476 - val_accuracy: 0.9910 - val_loss: 0.0514
Epoch 46/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9896 - loss: 0.0513 - val_accuracy: 0.9915 - val_loss: 0.0513
Epoch 47/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9893 - loss: 0.0469 - val_accuracy: 0.9915 - val_loss: 0.0511
Epoch 48/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9890 - loss: 0.0516 - val_accuracy: 0.9918 - val_loss: 0.0510
Epoch 49/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9894 - loss: 0.0505 - val_accuracy: 0.9910 - val_loss: 0.0505
Epoch 50/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9902 - loss: 0.0470 - val_accuracy: 0.9915 - val_loss: 0.0507
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
	Evaluation on Training

Accuracy       : 0.9902
F1 Score       : 0.9055
ROC AUC Score  : 0.9595
Precision      : 0.9803
Recall         : 0.8412

Confusion Matrix:
 [[15097    15]
 [  141   747]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      0.99     15112
           1       0.98      0.84      0.91       888

    accuracy                           0.99     16000
   macro avg       0.99      0.92      0.95     16000
weighted avg       0.99      0.99      0.99     16000

125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
	Evaluation on Validation

Accuracy       : 0.9915
F1 Score       : 0.9190
ROC AUC Score  : 0.9434
Precision      : 0.9747
Recall         : 0.8694

Confusion Matrix:
 [[3773    5]
 [  29  193]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      1.00      3778
           1       0.97      0.87      0.92       222

    accuracy                           0.99      4000
   macro avg       0.98      0.93      0.96      4000
weighted avg       0.99      0.99      0.99      4000

Observation:

  • This model has a better performance on the training and validation datasets with high accuracy values (0.989 on training and 0.988 on validation) and precision values (0.979 on training and 0.973 on validation).
  • The recall for class 1 is lower than that of class 0 and indicates that the model is good at predicting the majority class.
  • The f1-score for class 0 is higher than that of class 1 (0.99 on class 0 and 0.89 on class 1).
In [96]:
# model 2: Deeper NN + SGD
m2, h2 = build_model([64, 32, 16], SGD(learning_rate=0.01),
                     dropout_rate=0.1, X_tr=X_train_scaled, y_tr = y_train.values,
                     X_val = X_val_scaled, y_val = y_val.values,
                     use_class_weights=False, epochs=50, batch_size=64, verbose=1)
Training model: layers=[64, 32, 16], optimizer=str, dropout=0.1, class_weights=False
Epoch 1/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.8146 - loss: 0.4482 - val_accuracy: 0.9445 - val_loss: 0.2146
Epoch 2/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - accuracy: 0.9450 - loss: 0.2092 - val_accuracy: 0.9445 - val_loss: 0.1867
Epoch 3/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9435 - loss: 0.1815 - val_accuracy: 0.9450 - val_loss: 0.1669
Epoch 4/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9497 - loss: 0.1571 - val_accuracy: 0.9510 - val_loss: 0.1499
Epoch 5/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9538 - loss: 0.1497 - val_accuracy: 0.9578 - val_loss: 0.1366
Epoch 6/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9610 - loss: 0.1347 - val_accuracy: 0.9630 - val_loss: 0.1244
Epoch 7/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9648 - loss: 0.1264 - val_accuracy: 0.9697 - val_loss: 0.1144
Epoch 8/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9696 - loss: 0.1128 - val_accuracy: 0.9732 - val_loss: 0.1065
Epoch 9/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9694 - loss: 0.1143 - val_accuracy: 0.9753 - val_loss: 0.0997
Epoch 10/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9738 - loss: 0.0964 - val_accuracy: 0.9768 - val_loss: 0.0944
Epoch 11/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9710 - loss: 0.1054 - val_accuracy: 0.9775 - val_loss: 0.0896
Epoch 12/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9756 - loss: 0.0887 - val_accuracy: 0.9785 - val_loss: 0.0852
Epoch 13/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9755 - loss: 0.0935 - val_accuracy: 0.9787 - val_loss: 0.0825
Epoch 14/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9765 - loss: 0.0910 - val_accuracy: 0.9812 - val_loss: 0.0787
Epoch 15/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9804 - loss: 0.0809 - val_accuracy: 0.9822 - val_loss: 0.0758
Epoch 16/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9792 - loss: 0.0843 - val_accuracy: 0.9837 - val_loss: 0.0733
Epoch 17/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9804 - loss: 0.0761 - val_accuracy: 0.9830 - val_loss: 0.0718
Epoch 18/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9823 - loss: 0.0748 - val_accuracy: 0.9852 - val_loss: 0.0693
Epoch 19/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9823 - loss: 0.0778 - val_accuracy: 0.9862 - val_loss: 0.0676
Epoch 20/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9823 - loss: 0.0791 - val_accuracy: 0.9870 - val_loss: 0.0654
Epoch 21/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9848 - loss: 0.0643 - val_accuracy: 0.9877 - val_loss: 0.0640
Epoch 22/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9840 - loss: 0.0665 - val_accuracy: 0.9875 - val_loss: 0.0622
Epoch 23/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9837 - loss: 0.0717 - val_accuracy: 0.9883 - val_loss: 0.0610
Epoch 24/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9862 - loss: 0.0620 - val_accuracy: 0.9877 - val_loss: 0.0595
Epoch 25/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9863 - loss: 0.0644 - val_accuracy: 0.9877 - val_loss: 0.0585
Epoch 26/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9870 - loss: 0.0584 - val_accuracy: 0.9880 - val_loss: 0.0576
Epoch 27/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9859 - loss: 0.0612 - val_accuracy: 0.9872 - val_loss: 0.0568
Epoch 28/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9853 - loss: 0.0644 - val_accuracy: 0.9883 - val_loss: 0.0558
Epoch 29/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9871 - loss: 0.0557 - val_accuracy: 0.9890 - val_loss: 0.0552
Epoch 30/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9872 - loss: 0.0590 - val_accuracy: 0.9885 - val_loss: 0.0546
Epoch 31/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9865 - loss: 0.0583 - val_accuracy: 0.9893 - val_loss: 0.0537
Epoch 32/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9873 - loss: 0.0566 - val_accuracy: 0.9898 - val_loss: 0.0529
Epoch 33/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9891 - loss: 0.0502 - val_accuracy: 0.9895 - val_loss: 0.0524
Epoch 34/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9870 - loss: 0.0607 - val_accuracy: 0.9898 - val_loss: 0.0521
Epoch 35/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9863 - loss: 0.0607 - val_accuracy: 0.9898 - val_loss: 0.0516
Epoch 36/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9878 - loss: 0.0542 - val_accuracy: 0.9900 - val_loss: 0.0510
Epoch 37/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9881 - loss: 0.0530 - val_accuracy: 0.9898 - val_loss: 0.0509
Epoch 38/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9883 - loss: 0.0493 - val_accuracy: 0.9900 - val_loss: 0.0502
Epoch 39/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9886 - loss: 0.0495 - val_accuracy: 0.9902 - val_loss: 0.0503
Epoch 40/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9897 - loss: 0.0530 - val_accuracy: 0.9905 - val_loss: 0.0497
Epoch 41/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9875 - loss: 0.0510 - val_accuracy: 0.9900 - val_loss: 0.0493
Epoch 42/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9895 - loss: 0.0528 - val_accuracy: 0.9910 - val_loss: 0.0490
Epoch 43/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9901 - loss: 0.0480 - val_accuracy: 0.9910 - val_loss: 0.0487
Epoch 44/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9897 - loss: 0.0495 - val_accuracy: 0.9905 - val_loss: 0.0488
Epoch 45/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9885 - loss: 0.0560 - val_accuracy: 0.9910 - val_loss: 0.0483
Epoch 46/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9890 - loss: 0.0547 - val_accuracy: 0.9915 - val_loss: 0.0478
Epoch 47/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9892 - loss: 0.0536 - val_accuracy: 0.9915 - val_loss: 0.0475
Epoch 48/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9904 - loss: 0.0504 - val_accuracy: 0.9912 - val_loss: 0.0478
Epoch 49/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9895 - loss: 0.0531 - val_accuracy: 0.9905 - val_loss: 0.0471
Epoch 50/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9898 - loss: 0.0446 - val_accuracy: 0.9910 - val_loss: 0.0472
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
	Evaluation on Training

Accuracy       : 0.9912
F1 Score       : 0.9155
ROC AUC Score  : 0.9637
Precision      : 0.9870
Recall         : 0.8536

Confusion Matrix:
 [[15102    10]
 [  130   758]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      1.00     15112
           1       0.99      0.85      0.92       888

    accuracy                           0.99     16000
   macro avg       0.99      0.93      0.96     16000
weighted avg       0.99      0.99      0.99     16000

125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
	Evaluation on Validation

Accuracy       : 0.9910
F1 Score       : 0.9147
ROC AUC Score  : 0.9433
Precision      : 0.9650
Recall         : 0.8694

Confusion Matrix:
 [[3771    7]
 [  29  193]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      1.00      3778
           1       0.96      0.87      0.91       222

    accuracy                           0.99      4000
   macro avg       0.98      0.93      0.95      4000
weighted avg       0.99      0.99      0.99      4000

Observation:

  • This model has a better performance on the training and validation datasets with high accuracy values (0.991 on training and 0.992 on validation) and precision values (0.985 on training and 0.980 on validation).
  • The recall for class 1 is lower than that of class 0 and indicates that the model is good at predicting the majority class.
  • The f1-score for class 0 is higher than that of class 1 (0.99 on class 0 and 0.89 on class 1) and indicates that the model is better than the previous evaluation.
In [97]:
# model 3: NN + Adam Optimizer
m3, h3 = build_model([32, 16], SGD(learning_rate=0.001),
                     dropout_rate=0.0, X_tr=X_train_scaled, y_tr = y_train.values,
                     X_val = X_val_scaled, y_val = y_val.values,
                     use_class_weights=False, epochs=50, batch_size=64, verbose=1)
Training model: layers=[32, 16], optimizer=str, dropout=0.0, class_weights=False
Epoch 1/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.2420 - loss: 0.8965 - val_accuracy: 0.6768 - val_loss: 0.6210
Epoch 2/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7542 - loss: 0.5661 - val_accuracy: 0.8923 - val_loss: 0.4530
Epoch 3/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9117 - loss: 0.4229 - val_accuracy: 0.9365 - val_loss: 0.3644
Epoch 4/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9428 - loss: 0.3415 - val_accuracy: 0.9430 - val_loss: 0.3121
Epoch 5/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9453 - loss: 0.2968 - val_accuracy: 0.9445 - val_loss: 0.2793
Epoch 6/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9442 - loss: 0.2672 - val_accuracy: 0.9445 - val_loss: 0.2574
Epoch 7/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9424 - loss: 0.2499 - val_accuracy: 0.9445 - val_loss: 0.2421
Epoch 8/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9434 - loss: 0.2344 - val_accuracy: 0.9445 - val_loss: 0.2308
Epoch 9/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9461 - loss: 0.2209 - val_accuracy: 0.9445 - val_loss: 0.2220
Epoch 10/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9464 - loss: 0.2115 - val_accuracy: 0.9445 - val_loss: 0.2148
Epoch 11/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9451 - loss: 0.2060 - val_accuracy: 0.9445 - val_loss: 0.2086
Epoch 12/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9443 - loss: 0.2037 - val_accuracy: 0.9445 - val_loss: 0.2033
Epoch 13/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9465 - loss: 0.1921 - val_accuracy: 0.9445 - val_loss: 0.1985
Epoch 14/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9458 - loss: 0.1906 - val_accuracy: 0.9445 - val_loss: 0.1941
Epoch 15/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9465 - loss: 0.1833 - val_accuracy: 0.9445 - val_loss: 0.1901
Epoch 16/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9439 - loss: 0.1862 - val_accuracy: 0.9445 - val_loss: 0.1864
Epoch 17/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9408 - loss: 0.1877 - val_accuracy: 0.9445 - val_loss: 0.1829
Epoch 18/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9435 - loss: 0.1798 - val_accuracy: 0.9445 - val_loss: 0.1797
Epoch 19/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9451 - loss: 0.1729 - val_accuracy: 0.9448 - val_loss: 0.1767
Epoch 20/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9447 - loss: 0.1722 - val_accuracy: 0.9448 - val_loss: 0.1739
Epoch 21/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9461 - loss: 0.1689 - val_accuracy: 0.9448 - val_loss: 0.1712
Epoch 22/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9434 - loss: 0.1710 - val_accuracy: 0.9450 - val_loss: 0.1687
Epoch 23/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9453 - loss: 0.1597 - val_accuracy: 0.9452 - val_loss: 0.1662
Epoch 24/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9471 - loss: 0.1577 - val_accuracy: 0.9460 - val_loss: 0.1639
Epoch 25/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9449 - loss: 0.1586 - val_accuracy: 0.9463 - val_loss: 0.1617
Epoch 26/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9475 - loss: 0.1539 - val_accuracy: 0.9463 - val_loss: 0.1595
Epoch 27/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9452 - loss: 0.1565 - val_accuracy: 0.9470 - val_loss: 0.1574
Epoch 28/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9470 - loss: 0.1526 - val_accuracy: 0.9473 - val_loss: 0.1554
Epoch 29/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9465 - loss: 0.1527 - val_accuracy: 0.9475 - val_loss: 0.1535
Epoch 30/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9430 - loss: 0.1551 - val_accuracy: 0.9482 - val_loss: 0.1516
Epoch 31/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9493 - loss: 0.1490 - val_accuracy: 0.9492 - val_loss: 0.1497
Epoch 32/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9508 - loss: 0.1420 - val_accuracy: 0.9498 - val_loss: 0.1480
Epoch 33/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9511 - loss: 0.1476 - val_accuracy: 0.9505 - val_loss: 0.1462
Epoch 34/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9540 - loss: 0.1352 - val_accuracy: 0.9507 - val_loss: 0.1446
Epoch 35/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9531 - loss: 0.1400 - val_accuracy: 0.9513 - val_loss: 0.1429
Epoch 36/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9542 - loss: 0.1388 - val_accuracy: 0.9517 - val_loss: 0.1413
Epoch 37/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9578 - loss: 0.1346 - val_accuracy: 0.9528 - val_loss: 0.1398
Epoch 38/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9553 - loss: 0.1344 - val_accuracy: 0.9538 - val_loss: 0.1383
Epoch 39/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9557 - loss: 0.1332 - val_accuracy: 0.9555 - val_loss: 0.1368
Epoch 40/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9568 - loss: 0.1406 - val_accuracy: 0.9565 - val_loss: 0.1354
Epoch 41/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9597 - loss: 0.1310 - val_accuracy: 0.9578 - val_loss: 0.1340
Epoch 42/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9620 - loss: 0.1218 - val_accuracy: 0.9590 - val_loss: 0.1327
Epoch 43/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9586 - loss: 0.1328 - val_accuracy: 0.9595 - val_loss: 0.1314
Epoch 44/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9612 - loss: 0.1332 - val_accuracy: 0.9605 - val_loss: 0.1301
Epoch 45/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9633 - loss: 0.1208 - val_accuracy: 0.9610 - val_loss: 0.1289
Epoch 46/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9608 - loss: 0.1241 - val_accuracy: 0.9617 - val_loss: 0.1277
Epoch 47/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9642 - loss: 0.1233 - val_accuracy: 0.9630 - val_loss: 0.1266
Epoch 48/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9628 - loss: 0.1239 - val_accuracy: 0.9638 - val_loss: 0.1254
Epoch 49/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9625 - loss: 0.1260 - val_accuracy: 0.9650 - val_loss: 0.1244
Epoch 50/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9656 - loss: 0.1231 - val_accuracy: 0.9650 - val_loss: 0.1233
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step
	Evaluation on Training

Accuracy       : 0.9652
F1 Score       : 0.5460
ROC AUC Score  : 0.9179
Precision      : 0.9882
Recall         : 0.3773

Confusion Matrix:
 [[15108     4]
 [  553   335]]

Classification Report:
               precision    recall  f1-score   support

           0       0.96      1.00      0.98     15112
           1       0.99      0.38      0.55       888

    accuracy                           0.97     16000
   macro avg       0.98      0.69      0.76     16000
weighted avg       0.97      0.97      0.96     16000

125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
	Evaluation on Validation

Accuracy       : 0.9650
F1 Score       : 0.5395
ROC AUC Score  : 0.9111
Precision      : 1.0000
Recall         : 0.3694

Confusion Matrix:
 [[3778    0]
 [ 140   82]]

Classification Report:
               precision    recall  f1-score   support

           0       0.96      1.00      0.98      3778
           1       1.00      0.37      0.54       222

    accuracy                           0.96      4000
   macro avg       0.98      0.68      0.76      4000
weighted avg       0.97      0.96      0.96      4000

Observation:

  • This model has a better performance on the training and validation datasets with high accuracy values (0.968 on training and 0.970 on validation) and precision values (0.937 on training and 0.981 on validation).
  • The recall values for both the training and the validation is very low which thus indicates that the model is heavily biased towards the majority class (class 0).
  • The f1-score for class 0 is much higher than that of class 1 (0.98 on class 0 and 0.64 on class 1) and indicates that the model is missing more than half of the minority instances.
In [98]:
# model 4: NN + Adam + Dropout
m4, h4 = build_model([64, 32], SGD(learning_rate=0.001),
                     dropout_rate=0.3, X_tr=X_train_scaled, y_tr = y_train.values,
                     X_val = X_val_scaled, y_val = y_val.values,
                     use_class_weights=False, epochs=50, batch_size=64, verbose=1)
Training model: layers=[64, 32], optimizer=str, dropout=0.3, class_weights=False
Epoch 1/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.7279 - loss: 0.5717 - val_accuracy: 0.9420 - val_loss: 0.4068
Epoch 2/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9165 - loss: 0.3989 - val_accuracy: 0.9445 - val_loss: 0.3206
Epoch 3/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9406 - loss: 0.3197 - val_accuracy: 0.9445 - val_loss: 0.2794
Epoch 4/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9457 - loss: 0.2805 - val_accuracy: 0.9445 - val_loss: 0.2561
Epoch 5/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9463 - loss: 0.2638 - val_accuracy: 0.9445 - val_loss: 0.2408
Epoch 6/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9410 - loss: 0.2591 - val_accuracy: 0.9445 - val_loss: 0.2296
Epoch 7/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9449 - loss: 0.2395 - val_accuracy: 0.9445 - val_loss: 0.2207
Epoch 8/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9448 - loss: 0.2317 - val_accuracy: 0.9445 - val_loss: 0.2133
Epoch 9/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9437 - loss: 0.2226 - val_accuracy: 0.9445 - val_loss: 0.2070
Epoch 10/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9472 - loss: 0.2140 - val_accuracy: 0.9445 - val_loss: 0.2015
Epoch 11/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9471 - loss: 0.2115 - val_accuracy: 0.9445 - val_loss: 0.1966
Epoch 12/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9449 - loss: 0.2078 - val_accuracy: 0.9448 - val_loss: 0.1921
Epoch 13/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9488 - loss: 0.1959 - val_accuracy: 0.9448 - val_loss: 0.1881
Epoch 14/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9510 - loss: 0.1934 - val_accuracy: 0.9452 - val_loss: 0.1844
Epoch 15/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9500 - loss: 0.1867 - val_accuracy: 0.9460 - val_loss: 0.1810
Epoch 16/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9457 - loss: 0.1941 - val_accuracy: 0.9475 - val_loss: 0.1778
Epoch 17/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9540 - loss: 0.1806 - val_accuracy: 0.9477 - val_loss: 0.1749
Epoch 18/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9514 - loss: 0.1839 - val_accuracy: 0.9488 - val_loss: 0.1722
Epoch 19/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9514 - loss: 0.1761 - val_accuracy: 0.9498 - val_loss: 0.1696
Epoch 20/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9517 - loss: 0.1757 - val_accuracy: 0.9510 - val_loss: 0.1671
Epoch 21/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9545 - loss: 0.1684 - val_accuracy: 0.9520 - val_loss: 0.1648
Epoch 22/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9548 - loss: 0.1683 - val_accuracy: 0.9538 - val_loss: 0.1626
Epoch 23/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9543 - loss: 0.1672 - val_accuracy: 0.9545 - val_loss: 0.1604
Epoch 24/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9502 - loss: 0.1758 - val_accuracy: 0.9550 - val_loss: 0.1584
Epoch 25/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9540 - loss: 0.1678 - val_accuracy: 0.9563 - val_loss: 0.1564
Epoch 26/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9587 - loss: 0.1548 - val_accuracy: 0.9565 - val_loss: 0.1546
Epoch 27/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9519 - loss: 0.1686 - val_accuracy: 0.9570 - val_loss: 0.1528
Epoch 28/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9575 - loss: 0.1545 - val_accuracy: 0.9585 - val_loss: 0.1511
Epoch 29/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9533 - loss: 0.1607 - val_accuracy: 0.9590 - val_loss: 0.1494
Epoch 30/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9582 - loss: 0.1571 - val_accuracy: 0.9600 - val_loss: 0.1478
Epoch 31/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9587 - loss: 0.1546 - val_accuracy: 0.9600 - val_loss: 0.1462
Epoch 32/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9586 - loss: 0.1525 - val_accuracy: 0.9603 - val_loss: 0.1447
Epoch 33/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9581 - loss: 0.1514 - val_accuracy: 0.9613 - val_loss: 0.1433
Epoch 34/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9563 - loss: 0.1551 - val_accuracy: 0.9615 - val_loss: 0.1419
Epoch 35/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9593 - loss: 0.1548 - val_accuracy: 0.9617 - val_loss: 0.1405
Epoch 36/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9568 - loss: 0.1536 - val_accuracy: 0.9622 - val_loss: 0.1393
Epoch 37/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9592 - loss: 0.1468 - val_accuracy: 0.9625 - val_loss: 0.1380
Epoch 38/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9612 - loss: 0.1496 - val_accuracy: 0.9628 - val_loss: 0.1368
Epoch 39/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9614 - loss: 0.1471 - val_accuracy: 0.9630 - val_loss: 0.1356
Epoch 40/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9556 - loss: 0.1568 - val_accuracy: 0.9630 - val_loss: 0.1344
Epoch 41/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9603 - loss: 0.1446 - val_accuracy: 0.9630 - val_loss: 0.1333
Epoch 42/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9614 - loss: 0.1411 - val_accuracy: 0.9630 - val_loss: 0.1322
Epoch 43/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9602 - loss: 0.1420 - val_accuracy: 0.9632 - val_loss: 0.1312
Epoch 44/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9624 - loss: 0.1363 - val_accuracy: 0.9632 - val_loss: 0.1301
Epoch 45/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9602 - loss: 0.1398 - val_accuracy: 0.9635 - val_loss: 0.1292
Epoch 46/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9592 - loss: 0.1433 - val_accuracy: 0.9643 - val_loss: 0.1282
Epoch 47/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9621 - loss: 0.1384 - val_accuracy: 0.9643 - val_loss: 0.1273
Epoch 48/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9609 - loss: 0.1330 - val_accuracy: 0.9645 - val_loss: 0.1264
Epoch 49/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9634 - loss: 0.1330 - val_accuracy: 0.9645 - val_loss: 0.1256
Epoch 50/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9622 - loss: 0.1351 - val_accuracy: 0.9643 - val_loss: 0.1248
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step
	Evaluation on Training

Accuracy       : 0.9656
F1 Score       : 0.5567
ROC AUC Score  : 0.9054
Precision      : 0.9746
Recall         : 0.3896

Confusion Matrix:
 [[15103     9]
 [  542   346]]

Classification Report:
               precision    recall  f1-score   support

           0       0.97      1.00      0.98     15112
           1       0.97      0.39      0.56       888

    accuracy                           0.97     16000
   macro avg       0.97      0.69      0.77     16000
weighted avg       0.97      0.97      0.96     16000

125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
	Evaluation on Validation

Accuracy       : 0.9643
F1 Score       : 0.5342
ROC AUC Score  : 0.8955
Precision      : 0.9647
Recall         : 0.3694

Confusion Matrix:
 [[3775    3]
 [ 140   82]]

Classification Report:
               precision    recall  f1-score   support

           0       0.96      1.00      0.98      3778
           1       0.96      0.37      0.53       222

    accuracy                           0.96      4000
   macro avg       0.96      0.68      0.76      4000
weighted avg       0.96      0.96      0.96      4000

Observation:

  • This model has a better performance on the training and validation datasets with high accuracy values (0.966 on training and 0.963 on validation) and high precision values (0.985 on training and 0.975 on validation).
  • The recall values for both the training and the validation is very low (0.393 on training and 0.355 on validation) which thus indicates that it is much lower for the the minority class.
  • The f1-score for class 0 is much higher than that of class 1 (0.56 on training and 0.52 on validation) and indicates that the model is missing a large portion of true positive values.
In [99]:
# model 5: NN + SGD + Class Weight
m5, h5 = build_model([32, 16], SGD(learning_rate=0.01),
                     dropout_rate=0.0, X_tr=X_train_scaled, y_tr = y_train.values,
                     X_val = X_val_scaled, y_val = y_val.values,
                     use_class_weights=False, epochs=50, batch_size=64, verbose=1)
Training model: layers=[32, 16], optimizer=str, dropout=0.0, class_weights=False
Epoch 1/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.6116 - loss: 0.7249 - val_accuracy: 0.9445 - val_loss: 0.2285
Epoch 2/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9428 - loss: 0.2126 - val_accuracy: 0.9448 - val_loss: 0.1867
Epoch 3/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9475 - loss: 0.1705 - val_accuracy: 0.9520 - val_loss: 0.1614
Epoch 4/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9536 - loss: 0.1476 - val_accuracy: 0.9582 - val_loss: 0.1428
Epoch 5/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9616 - loss: 0.1311 - val_accuracy: 0.9645 - val_loss: 0.1294
Epoch 6/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9664 - loss: 0.1219 - val_accuracy: 0.9660 - val_loss: 0.1200
Epoch 7/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9698 - loss: 0.1130 - val_accuracy: 0.9682 - val_loss: 0.1125
Epoch 8/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9698 - loss: 0.1069 - val_accuracy: 0.9703 - val_loss: 0.1065
Epoch 9/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9756 - loss: 0.0945 - val_accuracy: 0.9722 - val_loss: 0.1012
Epoch 10/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9733 - loss: 0.0961 - val_accuracy: 0.9747 - val_loss: 0.0965
Epoch 11/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9739 - loss: 0.0961 - val_accuracy: 0.9755 - val_loss: 0.0922
Epoch 12/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9787 - loss: 0.0835 - val_accuracy: 0.9772 - val_loss: 0.0883
Epoch 13/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9790 - loss: 0.0829 - val_accuracy: 0.9785 - val_loss: 0.0847
Epoch 14/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9796 - loss: 0.0813 - val_accuracy: 0.9795 - val_loss: 0.0816
Epoch 15/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9801 - loss: 0.0773 - val_accuracy: 0.9805 - val_loss: 0.0787
Epoch 16/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9803 - loss: 0.0798 - val_accuracy: 0.9805 - val_loss: 0.0763
Epoch 17/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9831 - loss: 0.0702 - val_accuracy: 0.9812 - val_loss: 0.0740
Epoch 18/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9820 - loss: 0.0711 - val_accuracy: 0.9818 - val_loss: 0.0722
Epoch 19/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.9818 - loss: 0.0701 - val_accuracy: 0.9825 - val_loss: 0.0705
Epoch 20/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9824 - loss: 0.0698 - val_accuracy: 0.9833 - val_loss: 0.0690
Epoch 21/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9870 - loss: 0.0605 - val_accuracy: 0.9840 - val_loss: 0.0678
Epoch 22/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9837 - loss: 0.0682 - val_accuracy: 0.9843 - val_loss: 0.0663
Epoch 23/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9860 - loss: 0.0646 - val_accuracy: 0.9847 - val_loss: 0.0651
Epoch 24/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9846 - loss: 0.0690 - val_accuracy: 0.9850 - val_loss: 0.0641
Epoch 25/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9858 - loss: 0.0659 - val_accuracy: 0.9850 - val_loss: 0.0632
Epoch 26/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9862 - loss: 0.0626 - val_accuracy: 0.9852 - val_loss: 0.0623
Epoch 27/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9873 - loss: 0.0599 - val_accuracy: 0.9858 - val_loss: 0.0615
Epoch 28/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9870 - loss: 0.0627 - val_accuracy: 0.9855 - val_loss: 0.0608
Epoch 29/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9873 - loss: 0.0591 - val_accuracy: 0.9855 - val_loss: 0.0600
Epoch 30/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9858 - loss: 0.0648 - val_accuracy: 0.9865 - val_loss: 0.0592
Epoch 31/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9887 - loss: 0.0544 - val_accuracy: 0.9872 - val_loss: 0.0586
Epoch 32/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9881 - loss: 0.0533 - val_accuracy: 0.9877 - val_loss: 0.0582
Epoch 33/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9891 - loss: 0.0532 - val_accuracy: 0.9877 - val_loss: 0.0574
Epoch 34/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9877 - loss: 0.0589 - val_accuracy: 0.9880 - val_loss: 0.0568
Epoch 35/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9886 - loss: 0.0535 - val_accuracy: 0.9877 - val_loss: 0.0563
Epoch 36/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9879 - loss: 0.0585 - val_accuracy: 0.9880 - val_loss: 0.0559
Epoch 37/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9889 - loss: 0.0516 - val_accuracy: 0.9883 - val_loss: 0.0556
Epoch 38/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9880 - loss: 0.0572 - val_accuracy: 0.9883 - val_loss: 0.0549
Epoch 39/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9887 - loss: 0.0557 - val_accuracy: 0.9883 - val_loss: 0.0546
Epoch 40/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9883 - loss: 0.0558 - val_accuracy: 0.9883 - val_loss: 0.0540
Epoch 41/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9896 - loss: 0.0528 - val_accuracy: 0.9883 - val_loss: 0.0534
Epoch 42/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9896 - loss: 0.0488 - val_accuracy: 0.9895 - val_loss: 0.0534
Epoch 43/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9889 - loss: 0.0545 - val_accuracy: 0.9885 - val_loss: 0.0527
Epoch 44/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9882 - loss: 0.0512 - val_accuracy: 0.9885 - val_loss: 0.0522
Epoch 45/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9893 - loss: 0.0556 - val_accuracy: 0.9893 - val_loss: 0.0519
Epoch 46/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9887 - loss: 0.0542 - val_accuracy: 0.9895 - val_loss: 0.0515
Epoch 47/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9904 - loss: 0.0492 - val_accuracy: 0.9895 - val_loss: 0.0511
Epoch 48/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9895 - loss: 0.0509 - val_accuracy: 0.9895 - val_loss: 0.0508
Epoch 49/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9878 - loss: 0.0645 - val_accuracy: 0.9900 - val_loss: 0.0508
Epoch 50/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.9893 - loss: 0.0564 - val_accuracy: 0.9902 - val_loss: 0.0506
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
	Evaluation on Training

Accuracy       : 0.9899
F1 Score       : 0.9018
ROC AUC Score  : 0.9575
Precision      : 0.9840
Recall         : 0.8322

Confusion Matrix:
 [[15100    12]
 [  149   739]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      0.99     15112
           1       0.98      0.83      0.90       888

    accuracy                           0.99     16000
   macro avg       0.99      0.92      0.95     16000
weighted avg       0.99      0.99      0.99     16000

125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
	Evaluation on Validation

Accuracy       : 0.9902
F1 Score       : 0.9060
ROC AUC Score  : 0.9464
Precision      : 0.9741
Recall         : 0.8468

Confusion Matrix:
 [[3773    5]
 [  34  188]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      0.99      3778
           1       0.97      0.85      0.91       222

    accuracy                           0.99      4000
   macro avg       0.98      0.92      0.95      4000
weighted avg       0.99      0.99      0.99      4000

Observation:

  • This model has a better performance on the training and validation datasets with high accuracy values (0.989 on training and 0.989 on validation) and high precision values (0.981 on training and 0.973 on validation).
  • The recall values for both the training and the validation is very low (0.825 on training and 0.828 on validation) which thus indicates that it is much lower for the the minority class.
  • The f1-score for class 0 is much higher than that of class 1 (0.90) and indicates that the model is effectively detecting minority instances.
In [100]:
# model 6: Deep NN + Adam + Dropout + Class Weight
m6, h6 = build_model([128, 64, 32], Adam(learning_rate=0.001),
                     dropout_rate=0.4, X_tr=X_train_scaled, y_tr = y_train.values,
                     X_val = X_val_scaled, y_val = y_val.values,
                     use_class_weights=False, epochs=50, batch_size=64, verbose=1)
Training model: layers=[128, 64, 32], optimizer=str, dropout=0.4, class_weights=False
Epoch 1/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 3s 7ms/step - accuracy: 0.9388 - loss: 0.2210 - val_accuracy: 0.9835 - val_loss: 0.0817
Epoch 2/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9818 - loss: 0.0839 - val_accuracy: 0.9895 - val_loss: 0.0597
Epoch 3/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9856 - loss: 0.0706 - val_accuracy: 0.9920 - val_loss: 0.0510
Epoch 4/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9895 - loss: 0.0565 - val_accuracy: 0.9920 - val_loss: 0.0468
Epoch 5/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9882 - loss: 0.0617 - val_accuracy: 0.9927 - val_loss: 0.0435
Epoch 6/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9906 - loss: 0.0520 - val_accuracy: 0.9915 - val_loss: 0.0425
Epoch 7/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9919 - loss: 0.0494 - val_accuracy: 0.9920 - val_loss: 0.0414
Epoch 8/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9919 - loss: 0.0467 - val_accuracy: 0.9927 - val_loss: 0.0409
Epoch 9/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9912 - loss: 0.0494 - val_accuracy: 0.9920 - val_loss: 0.0426
Epoch 10/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9911 - loss: 0.0500 - val_accuracy: 0.9918 - val_loss: 0.0468
Epoch 11/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9908 - loss: 0.0532 - val_accuracy: 0.9927 - val_loss: 0.0386
Epoch 12/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9926 - loss: 0.0468 - val_accuracy: 0.9937 - val_loss: 0.0356
Epoch 13/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9926 - loss: 0.0360 - val_accuracy: 0.9930 - val_loss: 0.0373
Epoch 14/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9930 - loss: 0.0361 - val_accuracy: 0.9930 - val_loss: 0.0395
Epoch 15/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9920 - loss: 0.0406 - val_accuracy: 0.9927 - val_loss: 0.0381
Epoch 16/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9938 - loss: 0.0340 - val_accuracy: 0.9930 - val_loss: 0.0406
Epoch 17/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9930 - loss: 0.0357 - val_accuracy: 0.9935 - val_loss: 0.0374
Epoch 18/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - accuracy: 0.9924 - loss: 0.0367 - val_accuracy: 0.9935 - val_loss: 0.0381
Epoch 19/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9925 - loss: 0.0397 - val_accuracy: 0.9927 - val_loss: 0.0413
Epoch 20/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9924 - loss: 0.0397 - val_accuracy: 0.9930 - val_loss: 0.0378
Epoch 21/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9934 - loss: 0.0311 - val_accuracy: 0.9933 - val_loss: 0.0386
Epoch 22/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9933 - loss: 0.0353 - val_accuracy: 0.9923 - val_loss: 0.0415
Epoch 23/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9924 - loss: 0.0383 - val_accuracy: 0.9908 - val_loss: 0.0454
Epoch 24/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9933 - loss: 0.0340 - val_accuracy: 0.9918 - val_loss: 0.0424
Epoch 25/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9930 - loss: 0.0390 - val_accuracy: 0.9930 - val_loss: 0.0397
Epoch 26/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9931 - loss: 0.0345 - val_accuracy: 0.9925 - val_loss: 0.0413
Epoch 27/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9941 - loss: 0.0285 - val_accuracy: 0.9910 - val_loss: 0.0439
Epoch 28/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9929 - loss: 0.0361 - val_accuracy: 0.9915 - val_loss: 0.0414
Epoch 29/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9941 - loss: 0.0301 - val_accuracy: 0.9930 - val_loss: 0.0403
Epoch 30/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9942 - loss: 0.0334 - val_accuracy: 0.9933 - val_loss: 0.0437
Epoch 31/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9940 - loss: 0.0342 - val_accuracy: 0.9920 - val_loss: 0.0418
Epoch 32/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9925 - loss: 0.0366 - val_accuracy: 0.9927 - val_loss: 0.0413
Epoch 33/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9929 - loss: 0.0361 - val_accuracy: 0.9933 - val_loss: 0.0415
Epoch 34/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9938 - loss: 0.0320 - val_accuracy: 0.9925 - val_loss: 0.0411
Epoch 35/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9932 - loss: 0.0329 - val_accuracy: 0.9912 - val_loss: 0.0461
Epoch 36/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9934 - loss: 0.0327 - val_accuracy: 0.9930 - val_loss: 0.0431
Epoch 37/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9928 - loss: 0.0317 - val_accuracy: 0.9927 - val_loss: 0.0461
Epoch 38/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9945 - loss: 0.0323 - val_accuracy: 0.9905 - val_loss: 0.0472
Epoch 39/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9931 - loss: 0.0309 - val_accuracy: 0.9925 - val_loss: 0.0445
Epoch 40/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9941 - loss: 0.0279 - val_accuracy: 0.9933 - val_loss: 0.0444
Epoch 41/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9932 - loss: 0.0290 - val_accuracy: 0.9925 - val_loss: 0.0447
Epoch 42/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9945 - loss: 0.0278 - val_accuracy: 0.9930 - val_loss: 0.0477
Epoch 43/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9933 - loss: 0.0317 - val_accuracy: 0.9918 - val_loss: 0.0488
Epoch 44/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9935 - loss: 0.0293 - val_accuracy: 0.9923 - val_loss: 0.0472
Epoch 45/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9939 - loss: 0.0274 - val_accuracy: 0.9920 - val_loss: 0.0528
Epoch 46/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9943 - loss: 0.0292 - val_accuracy: 0.9925 - val_loss: 0.0475
Epoch 47/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9943 - loss: 0.0264 - val_accuracy: 0.9908 - val_loss: 0.0539
Epoch 48/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9949 - loss: 0.0265 - val_accuracy: 0.9933 - val_loss: 0.0455
Epoch 49/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9939 - loss: 0.0294 - val_accuracy: 0.9920 - val_loss: 0.0521
Epoch 50/50
250/250 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9936 - loss: 0.0293 - val_accuracy: 0.9912 - val_loss: 0.0526
500/500 ━━━━━━━━━━━━━━━━━━━━ 1s 1ms/step
	Evaluation on Training

Accuracy       : 0.9947
F1 Score       : 0.9505
ROC AUC Score  : 0.9965
Precision      : 0.9843
Recall         : 0.9189

Confusion Matrix:
 [[15099    13]
 [   72   816]]

Classification Report:
               precision    recall  f1-score   support

           0       1.00      1.00      1.00     15112
           1       0.98      0.92      0.95       888

    accuracy                           0.99     16000
   macro avg       0.99      0.96      0.97     16000
weighted avg       0.99      0.99      0.99     16000

125/125 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step
	Evaluation on Validation

Accuracy       : 0.9912
F1 Score       : 0.9203
ROC AUC Score  : 0.9666
Precision      : 0.9309
Recall         : 0.9099

Confusion Matrix:
 [[3763   15]
 [  20  202]]

Classification Report:
               precision    recall  f1-score   support

           0       0.99      1.00      1.00      3778
           1       0.93      0.91      0.92       222

    accuracy                           0.99      4000
   macro avg       0.96      0.95      0.96      4000
weighted avg       0.99      0.99      0.99      4000

Observation:

  • This model has a better performance on the training and validation datasets with high accuracy values (0.995 on training and 0.993 on validation) and high precision values (0.981 on training and 0.973 on validation).
  • The recall values for both the training and the validation is very low (0.917 on training and 0.914 on validation) which thus indicates that it is much lower for the the minority class.
  • The model has strong f1-score for class 0 is much higher than that of class 1 (0.96) and indicates that the model is effectively detecting minority instances.

Choosing the best model

  • As per the observations from Models 1 to 6, the best model choosen is Model 6 (Deep NN + Adam + Dropout + Class Weight).
  • Model 6 achieves the best balanced value for all the evaluation metrics.
  • It ensures both majority and minority classes are well predicted.
  • This model has the highest overall accuracy values, strong F1 score, high recall values and high precision for minority class.
  • With a good ROC-AUC score, it has a strong disccriminative ability between the classes.
  • The confusion matrix shows very low rates of false negatives and false positives demonstrating its reliability.

Actionable Insights & Recommendations

  • The dataset of ReneWind is highly imbalanced (90:10 ratio) and requires resampling or class weighting to ensure that the dataset becomes balanced for both the class 0 and class 1.
  • Several features have significant outliers which may affect the predictions if they are left untreated.
  • The precision for all the models are high and indicates that the false positives (FP) are minimal.
  • Each of the prediction could be translated as:
    • True positives (TP) are failures correctly predicted by the model which will result in repair costs.* False negatives (FN) are real failures where there is no detection by the model. These will result in replacement costs.
    • False positives (FP) are detections where there is no failure and will result in inspection costs.
  • Among all the 6 models that were tested, the Deep Neural Network with Adam optimizer, dropout and class weights has provided the most balanced performance by achieving high accuracy.

Key Takeaways for the Business

  • The traditional machine learning methods would likely underperform because the failure patterns are not noticeable and non-linear. Thus, deep learning model provies the best fit.
  • The low correlation value of the data signalled to the turbine failures.
  • The turbine replacements are expensive when compared to inspections and thus every false negative (FN) helps the company from turbine replacement.
  • The model 6 (the Deep Neural Network with Adam optimizer, dropout and class weights model) provided high accuracy, recall and precision values, proving that it is reliable for deployment in operational environments.
  • This predictive approach helps in predicting the failures and thus the turbines can undergo planned maintenance instead of expensive downtime and replacements.

Conclusion

  • The dataset is anaysed and the outliers, imbalance and distribution of the features are found.
  • Several neural networks were tested from baseline neural network to deep neural network and the best model is found to be the Deep Neural Network with Adam optimizer, dropout and class weights.
  • This approach has reduced the false negatives and has resulted in minimizing replacement costs.
  • The deep learning based prediction aligns with the cost reduction and operational efficiency goals of ReneWind.
  • The predicted solution is generalizable and can be integrated to ral-time systems.

Business Recommendations

  • The Deep Neural Network with Adam optimizer, dropout and class weights model is found to have better performance when compared to the other models and hence this model can be integrated into the turbine monitoring platform.
  • The connection of temperature sensor and vibration sensor can help in feeding live data into the predictive model.
  • The predictive insights can be shared to the manufacturers to develop better turbines which can be monitored easily and has reduced downtime.
  • The predictive maintenance and preventive maintenance schedules can be combined to create a hybrid system to ensure that the turbines are inspected regularly.
  • A dashboard can be created for the operation and maintenance managers to visualize the real-time turbine conditions to feed live data into the predictive model.